Home appliance and method for controlling thereof

ABSTRACT

A home appliance is provided. The home appliance includes a sensor, a microphone, a speaker, and a processor. The processor is configured to, based on one of a first event wherein a user action is detected through the sensor or a second event wherein a trigger command for initiating a voice recognition mode is input through the microphone occurring, operate in the voice recognition mode, and control the speaker to output an audio signal corresponding to the event occurred, and the audio signal is an audio signal set differently for each of the first event and the second event.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.16/859,330, filed on Apr. 27, 2020, which is based on and claimspriority under 35 U.S.C. § 119(a) of Korean patent application No.10-2019-0050246, filed on Apr. 30, 2019, in the Korean IntellectualProperty Office, the disclosures of which are incorporated by referenceherein in their entireties.

BACKGROUND 1. Field

The disclosure relates to a home appliance and a method for controllingthereof, and more particularly, to a home appliance which operates in avoice recognition mode if an event that a user action is detectedoccurs, and a method for controlling thereof.

2. Description of Related Art

A home appliance can perform various functions according to a controlcommand of a user. For recent home appliances, not only a function ofreceiving input of a control command through input devices such as akeypad and a remote control, but also a voice recognition function ofreceiving input of a control command through a user voice is beingadopted.

In particular, there is a growing number of home appliances which, if auser utters a trigger command initiating a voice recognition function(e.g., Bixby), detect the trigger command and operate in a voicerecognition mode.

SUMMARY

The purpose of the disclosure is in providing a home appliance whichoperates in a voice recognition mode if an event that a user action isdetected occurs, and a method for controlling thereof.

A home appliance according to an embodiment of the disclosure includes asensor, a microphone, a speaker, and a processor. The processor may,based on one of a first event wherein a user action is detected throughthe sensor or a second event wherein a trigger command for initiating avoice recognition mode is input through the microphone occurring,operate in the voice recognition mode, and control the speaker to outputan audio signal corresponding to the event occurred, and the audiosignal may be an audio signal set differently for each of the firstevent and the second event.

Meanwhile, a method for controlling a home appliance according to anembodiment of the disclosure includes the steps of, based on one of afirst event wherein a user action is detected or a second event whereina trigger command for initiating a voice recognition mode is inputoccurring, outputting an audio signal corresponding to the eventoccurred, and operating in the voice recognition mode. Also, the audiosignal may be an audio signal set differently for each of the firstevent and the second event.

Meanwhile, in a recording medium storing a program for executing amethod for controlling a home appliance according to an embodiment ofthe disclosure, the controlling method includes the steps of, based onone of a first event wherein a user action is detected or a second eventwherein a trigger command for initiating a voice recognition mode isinput occurring, outputting an audio signal corresponding to the eventoccurred, and operating in the voice recognition mode. Also, the audiosignal may be an audio signal set differently for each of the firstevent and the second event.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram for illustrating a voice recognition systemaccording to an embodiment of the disclosure;

FIG. 2 is a block diagram illustrating a schematic configuration of ahome appliance according to an embodiment of the disclosure;

FIG. 3 is a block diagram illustrating a detailed configuration of ahome appliance according to an embodiment of the disclosure;

FIG. 4 is a block diagram illustrating a voice recognition moduleaccording to an embodiment of the disclosure;

FIG. 5 is a diagram for illustrating a method for initiating a voicerecognition mode according to the first embodiment of the disclosure;

FIG. 6 is a diagram for illustrating a method for initiating a voicerecognition mode according to the second embodiment of the disclosure;

FIG. 7 is a diagram for illustrating a method for converting to astandby mode according to the first embodiment of the disclosure;

FIG. 8 is a diagram for illustrating a method for converting to astandby mode according to the second embodiment of the disclosure;

FIG. 9 is a diagram illustrating examples of authority information ofeach of a plurality of users;

FIG. 10 is a diagram for illustrating execution of operations accordingto authority information of each of a plurality of users;

FIG. 11 is a diagram for illustrating an algorithm of a voicerecognition mode according to an embodiment of the disclosure;

FIG. 12 is a sequence diagram for illustrating execution of operationsaccording to authority information of each of a plurality of usersaccording to the first embodiment of the disclosure;

FIG. 13 is a sequence diagram for illustrating execution of operationsaccording to authority information of each of a plurality of usersaccording to the second embodiment of the disclosure; and

FIG. 14 is a flow chart for illustrating a method for controlling a homeappliance according to an embodiment of the disclosure.

DETAILED DESCRIPTION

First, terms used in this specification will be described briefly, andthen the disclosure will be described in detail.

As terms used in the embodiments of the disclosure, general terms thatare currently used widely were selected as far as possible, inconsideration of the functions described in the disclosure. However, theterms may vary depending on the intention of those skilled in the artwho work in the pertinent field, previous court decisions, or emergenceof new technologies. Also, in particular cases, there may be terms thatwere designated by the applicant on his own, and in such cases, themeaning of the terms will be described in detail in the relevantdescriptions in the disclosure. Thus, the terms used in the disclosureshould be defined based on the meaning of the terms and the overallcontent of the disclosure, but not just based on the names of the terms.

Also, various modifications may be made to the embodiments of thedisclosure, and there may be various types of embodiments. Accordingly,specific embodiments will be illustrated in drawings, and theembodiments will be described in detail in the detailed description.However, it should be noted that the various embodiments are not forlimiting the scope of the disclosure to a specific embodiment, but theyshould be interpreted to include all modifications, equivalents, oralternatives of the ideas and the technical scopes disclosed herein.Meanwhile, in case it is determined that in describing embodiments,detailed explanation of related known technologies may unnecessarilyconfuse the gist of the disclosure, the detailed explanation will beomitted.

In addition, terms such as “first,” “second,” and the like may be usedto describe various elements, but the terms are not intended to limitthe elements. Such terms are used only to distinguish one element fromanother element.

Meanwhile, singular expressions include plural expressions, unlessdefined obviously differently in the context. Further, in thedisclosure, terms such as “include” and “have” should be construed asdesignating that there are such characteristics, numbers, steps,operations, elements, components, or a combination thereof described inthe specification, but not as excluding in advance the existence orpossibility of adding one or more of other characteristics, numbers,steps, operations, elements, components, or a combination thereof.

Hereinafter, the embodiments of the disclosure will be described indetail with reference to the accompanying drawings, such that thosehaving ordinary skill in the art to which the disclosure belongs caneasily carry out the disclosure. However, it should be noted that thedisclosure may be implemented in various different forms, and is notlimited to the embodiments described herein. Also, in the drawings,parts that are not related to explanation were omitted, for explainingthe disclosure clearly.

Hereinafter, the disclosure will be described in more detail withreference to the drawings.

FIG. 1 is a diagram for illustrating a voice recognition systemaccording to an embodiment of the disclosure.

Referring to FIG. 1, the voice recognition system 1000 includes a homeappliance 100 and a server 200.

The home appliance 100 may be a refrigerator, a washing machine, anoven, a microwave oven, etc., and it may be a different device otherthan the aforementioned devices as well as the aforementioned devices,only if it supports a voice recognition mode.

The home appliance 100 may detect a wake-up action {circle around (1)}.Here, a wake-up action means an event wherein there is a possibilitythat a user utters a control command, an inquiry, or a response. Forexample, utterance of a trigger command by a user or opening and closingof the door of the home appliance 100, etc. may fall under a wake-upaction.

Also, the home appliance 100 may operate in a voice recognition mode{circle around (2)}. Then, the home appliance 100 may perform a feedbackinforming that the voice recognition mode was initiated {circle around(3)}. Specifically, the home appliance 100 may output an audio signalinforming that the voice recognition mode was initiated.

Here, an audio signal may be set differently according to the eventoccurred. Specifically, in case an event wherein a trigger command wasdetected occurred, a simple alarming sound informing that the voicerecognition mode was initiated may be output, and in case an eventwherein a user action was detected occurred, a voice messagespecifically informing that the voice recognition mode was initiated maybe output.

Then, the home appliance 100 may receive input of a user voice utteredby the user {circle around (4)}. Also, the home appliance 100 maytransmit the input user voice to the server 200 {circle around (5)}.

The server 200 means a computer (server computer) or a program (serverprogram) providing information or services to a client through anetwork.

Then, the server 200 may perform voice recognition for the input voice{circle around (6)}. Specifically, the server 200 may perform voicerecognition for the input voice by using a voice recognition module.Then, the server 200 may transmit the result of voice recognition to thehome appliance {circle around (7)}.

Then, the home appliance 100 may perform an operation corresponding tothe result of voice recognition. Meanwhile, before performing anoperation, the home appliance 100 may check to the user who uttered thevoice whether the user has authority for the operation, and if authorityis checked, the home appliance 100 may perform the operation. A detailedoperation in this regard will be described later with reference to FIG.3.

Meanwhile, in illustrating and describing FIG. 1, it was illustrated anddescribed that the voice recognition system includes one home applianceand one server, but in actual implementation, the voice recognitionsystem may be implemented to include a plurality of home appliances or aplurality of servers.

Also, in illustrating and describing FIG. 1, it was illustrated anddescribed that the voice recognition system includes a server, but inactual implementation, in case the home appliance is designed to be ableto perform all functions of a server, the voice recognition system maybe implemented to not include a server.

FIG. 2 is a block diagram illustrating a schematic configuration of ahome appliance according to an embodiment of the disclosure.

Referring to FIG. 2, the home appliance 100 includes a sensor 110, amicrophone 120, a processor 140, and a speaker 130.

The sensor 110 detects a user action. Specifically, the sensor 110 maydetect a user action for the home appliance 100.

For example, in case the sensor 110 is a sensor detecting opening andclosing of the door (not shown) provided on the main body of the homeappliance 100, the sensor 110 may detect opening or closing of the doorby the user and output a corresponding value. In this case, the sensor110 may be implemented as a switch, and if the switch is turned on,output a value corresponding to closing of the door, and if the switchis turned off, output a value corresponding to opening of the door.

As another example, in case the sensor 110 is a motion detection sensor,the sensor 110 may detect a user access for the home appliance 100. Inthis case, the sensor 110 may be implemented as an IR sensor, aproximity sensor, a thermal image sensor, or a camera, etc. As stillanother example, in case the sensor 110 is a touch detection sensor, thesensor 110 may detect a user touch for the home appliance 100.

Meanwhile, the types of the sensor 110 are not limited to theaforementioned examples, and may be different according to the type ormodel of the home appliance 100. Also, there may be a plurality ofsensors 110.

The microphone 120 may receive input of a voice. Specifically, themicrophone 120 may receive input of a voice uttered by a user andconvert the voice to voice data that can be processed at the homeappliance 100. Then, the microphone 120 may transmit the converted voicedata to the processor 140.

Meanwhile, in the illustrated embodiment, it was described that themicrophone 120 is installed inside the home appliance 100, but in actualimplementation, the microphone may be implemented as an external deviceof the home appliance. That is, the microphone may be implemented as aform of receiving a voice signal from an external microphone and usingthe voice signal.

The speaker 130 may be provided inside the home appliance 100 and outputa sound to the outside of the home appliance 100. Specifically, thespeaker 130 may convert an audio signal to a sound according to controlof the processor 140 and output the sound.

The processor 140 controls the home appliance 100. Specifically, theprocessor 140 may control each component of the home appliance 100according to a control command of a user. For example, in case the homeappliance 100 is a washing machine, if the processor 140 receives aspin-drying command, the processor 140 may control the operation of themotor to provide rotational force to the drum accommodating laundry.

Also, the processor 140 may operate in a voice recognition mode. Here, avoice recognition mode means an operation mode wherein the processor 140processes a user's inquiry and response input through the microphone 120as a natural language. Here, the processor 140 may be implemented in amanner of directly processing a user's inquiry and response as a naturallanguage by using a voice recognition module. Also, the processor 140may be implemented in a manner of transmitting a voice input through themicrophone 120 to the server and receiving the result of voicerecognition from the server. Meanwhile, detailed explanation regardingthe voice recognition module will be made later with reference to FIG.4.

Specifically, if a predetermined event occurs, the processor 140 mayoperate in the voice recognition mode. Here, a predetermined event meansan event wherein there is a possibility that a user utters a controlcommand, an inquiry, or a response.

For example, a predetermined event may be an event wherein a user uttersa trigger command for initiating the voice recognition mode, and thevoice including the trigger command is input through the microphone 120.In this case, the processor 140 may identify whether a trigger commandis included in voice data acquired through the microphone 120 by using atrigger command detection model, and if it is identified that a triggercommand is included, the processor 140 may operate in the voicerecognition mode. Meanwhile, a trigger command may be referred to as awake-up command or a starting command, etc.

As another example, a predetermined event may be an event wherein a useraction is detected through the sensor 110. In this case, if a useraction is detected through the sensor 110, the processor 140 may operatein the voice recognition mode. Here, a user action may be setdifferently according to the type of the sensor 110.

If the sensor 110 is a sensor for detecting opening and closing of thedoor provided on the main body of the home appliance 100, when openingor closing of the door is detected through the sensor 110, the processor140 may operate in the voice recognition mode. Meanwhile, the processor140 may be implemented in a manner of operating in the voice recognitionmode only when opening of the door is detected through the sensor 110,or in a manner of operating in the voice recognition mode only whenclosing of the door is detected. A detailed operation in this regardwill be described later with reference to FIGS. 5 and 6.

In contrast, in case the sensor 110 is a motion detection sensor, theprocessor 140 may operate in the voice recognition mode if a user accessfor the home appliance 100 is detected through the sensor 110.

As described above, if a trigger command for initiating the voicerecognition mode is input through the microphone 120 or a user action isdetected through the sensor 110, the processor 140 may operate in thevoice recognition mode.

Also, when the processor 140 operates in the voice recognition mode, theprocessor 140 may control the speaker 130 to output an audio signalinforming that the processor 140 operates in the voice recognition mode.

Specifically, when a predetermined event occurs and the processor 140operates in the voice recognition mode, the processor 140 may controlthe speaker 130 to output audio signals corresponding to eachpredetermined event.

To be more specific, if a trigger command is input through themicrophone 120 and the processor 140 operates in the voice recognitionmode, the processor 140 may control the speaker 130 to output a firstaudio signal. Also, if a user action is detected through the sensor 110and the processor 140 operates in the voice recognition mode, theprocessor 140 may control the speaker 130 to output a second signaldifferent from the first audio signal.

Here, the first audio signal may be a simple alarming sound. In case auser uttered a trigger command, the user intended the voice recognitionmode of the home appliance 100, and thus the alarming sound is forsimply informing that the home appliance 100 operates in the voicerecognition mode, and at the same time, for inducing the user tocontinuously utter a control command or an inquiry immediately withoutwaiting.

Meanwhile, the second audio signal may be a voice message informing thatthe voice recognition mode was initiated (e.g., “What can I help youwith?”). This is for specifically informing that the home appliance 100operates in the voice recognition mode and at the same time, forinducing the user to utter a control command or an inquiry, as it isuncertain if the user intended the voice recognition mode of the homeappliance 100 even if a user action is detected.

Then, the processor 140 may perform voice recognition for the voiceinput through the microphone 120. Specifically, after the processor 140started to operate in the voice recognition mode, the processor 140 mayperform voice recognition for the voice input through the microphone120.

Here, the processor 140 may perform voice recognition by using the voicerecognition module provided on the home appliance 100. Alternatively,the processor 140 may perform voice recognition in a manner oftransmitting voice data corresponding to the input voice to an externalserver (not shown), and receiving the result of voice recognition forthe transmitted voice data.

Then, the processor 140 may perform an operation corresponding to theresult of voice recognition. For example, in case the home appliance 100is a washing machine, and the result of voice recognition is “washingfor 30 minutes,” the processor 140 may control the home appliance 100 toperform a washing operation for 30 minutes.

In the above, only the schematic configuration of the home appliance 100was described, but the home appliance 100 may additionally include thecomponents as illustrated in FIG. 3. The detailed configuration of thehome appliance 100 will be described below with reference to FIG. 3.

Referring to FIG. 3, it is a block diagram illustrating a detailedconfiguration of a home appliance according to an embodiment of thedisclosure.

Referring to FIG. 3, the home appliance 100 according to an embodimentof the disclosure may consist of a sensor 110, a microphone 120, aspeaker 130, a processor 140, a user interface device 150, a memory 160,a communication device 170, and a display 180.

As the configurations of the sensor 110, the microphone 120, and thespeaker 130 are identical to the configurations in FIG. 2, overlappingexplanation will be omitted. Also, as the processor 140 was describedwith reference to FIG. 2, the contents described in FIG. 2 will not berepetitively described, but only the contents related to the componentsadded in FIG. 3 will be described below.

The processor 140 may include a plurality of processors 141, 143.Specifically, the processor 140 may include a first processor 141 fordetecting a trigger command and a second processor 143 for operating inthe voice recognition mode.

The first processor 141 may maintain a turned-on state in a standbystate. Here, a standby state means a state wherein the first processor141 detects whether a predetermined event occurred to initiate anoperation in the voice recognition mode. Also, when a user voiceincluding a trigger command is input through the microphone 120, thefirst processor 141 may output an alarming signal to the secondprocessor 143.

The first processor 141 as described above may be included in an audiochipset (not shown) that converts an analog acoustic signal to a digitalacoustic signal or converts a digital acoustic signal to an analogacoustic signal.

Meanwhile, the second processor 143 may maintain a turned-off state in astandby state. Specifically, when a standby state is maintained for overa predetermined time period, the state of the second processor 143 maybe changed from a turned-on state to a turned-off state and the statemay be maintained. Also, when the second processor 143 receives analarming signal output from the first processor 141, the secondprocessor 143 may be turned on and operate in the voice recognitionmode.

The user interface device 150 is a component for receiving input of auser interaction such as a user manipulation. Specifically, the userinterface device 150 may receive input of various control commands forthe home appliance 100 from a user. For example, in case the homeappliance 100 is a washing machine, the user interface device 150 mayinput a spin-drying command of the washing machine from a user.

The user interface device 150 may include buttons formed in any areas ofthe home appliance 100 such as the front surface, side surface, or rearsurface of the exterior of the main body or a light reception devicethat receives a light signal corresponding to a user input (e.g., atouch, a push, a touch gesture, a voice, or a motion) from a remotecontrol device. Meanwhile, if the display 180 is a touch screen, thedisplay 180 may also operate as the user interface device 150.

The user interface device 150 may receive input of a decision regardingwhether to activate each predetermined event from a user.

Specifically, the user interface device 150 may receive input of adecision regarding whether to activate each of an event wherein atrigger command for initiating the voice recognition mode is inputthrough the microphone 120 or an event wherein a user action is detectedthrough the sensor 110.

Then, when a predetermined event occurs, the processor 140 may operatein the voice recognition mode according to the result of the input.Specifically, in case the processor 140 received input of activation ofa predetermined event, the processor 140 may operate in the voicerecognition mode when the predetermined occurs. However, in case theprocessor 140 received input of inactivation of a predetermined event,the processor 140 may not operate in the voice recognition mode even ifthe predetermined event occurs.

For example, in case the user interface device 150 received input ofinactivation of an event wherein a trigger command for initiating thevoice recognition mode is input through the microphone 120, theprocessor 140 may not operate in the voice recognition mode even if atrigger command is input through the microphone 120.

As another example, in case the user interface device 150 received inputof inactivation of an event wherein a user action is detected throughthe sensor 110, the processor 140 may not operate in the voicerecognition mode even if a user action is detected through the sensor110.

Also, if inactivation of a predetermined event is input, the processor140 may perform an additional operation according to what theinactivated predetermined event is.

Specifically, if inactivation of an event wherein a trigger command forinitiating the voice recognition mode is input through the microphone120 is input, the processor 140 does not need to perform detection ofthe trigger command. Accordingly, the processor 140 may reduce powerconsumption by turning off the microphone 120 in the standby state ofthe home appliance 100.

Here, in case activation is executed for an event wherein a user actionis detected through the sensor 110, if a user action is detected throughthe sensor 110, the processor 140 may turn on the microphone 120 againfor an operation in the voice recognition mode, and execute the voicerecognition module.

In contrast, in case inactivation of an event wherein a user action isdetected through the sensor 110 is input, the processor 140 does notneed to detect a user action. Accordingly, the processor 140 may reducepower consumption by turning off the sensor 110 detecting a user actionin the standby state.

Here, in case activation is executed for an event wherein a triggercommand is input through the microphone 120, the processor 140 maymaintain a turned-on state of the microphone 120 in the standby statefor detection of a trigger command.

The memory 160 stores various kinds of data for the overall operationsof the home appliance 100 such as programs for processing or control ofthe processor 140.

Specifically, the memory 160 may store several application programsdriven in the home appliance 100, and data and commands for theoperations of the home appliance 100.

Also, the memory 160 may be accessed by the processor 140, andreading/recording/correction/deletion/update, etc. of data by theprocessor 140 may be performed. Such a memory 160 may be implemented notonly as a storage medium inside the home appliance 100, but also as anexternal storage medium, a removable disk including a USB memory, a webserver through a network, etc.

In addition, the memory 160 may store a voice recognition module. Also,the processor 140 may use the voice recognition module for generating anatural language as a response to a user utterance. The voicerecognition module may be executed by a conventional generic-purposeprocessor (e.g., a CPU) or a separate AI-dedicated processor (e.g., aGPU, etc.). Explanation in this regard will be made later with referenceto FIG. 4.

Further, the memory 160 may store a speaker recognition model. Here, aspeaker recognition model means a model which uses vocal printinformation included in a voice input through the microphone 120 andvocal print information of a pre-registered user, and identifies a usercorresponding to the current speaker among pre-registered users.

Then, the processor 140 may perform a speaker recognition function ofidentifying a user corresponding to the voice input through themicrophone 120 by using the speaker recognition model stored in thememory 160.

The memory 160 may store authority information of each user.Specifically, the memory 160 may store authority information for each ofa plurality of operations of the home appliance 100 by each of aplurality of users.

Then, if a user voice is input through the microphone 120, the processor140 may identify the user voice and recognize the user, and if theoperation of the home appliance 100 corresponding to the user voicematches the authority information of the recognized user, the processor140 may perform the operation of the home appliance 100 corresponding tothe user voice.

To be more specific, the processor 140 may recognize a user by using auser voice input through the microphone 120 in the voice recognitionmode and the speaker recognition model stored in the memory 160. Then,the processor 140 may identify whether the operation of the homeappliance 100 (e.g., a washing operation) corresponding to the uservoice matches the authority information of the recognized user by usingthe authority information of the user stored in the memory 160, and ifthe operation matches the authority, the processor 140 may perform theoperation of the home appliance 100 corresponding to the user voice.

Meanwhile, in case a user corresponding to a user voice input throughthe microphone 120 is not recognized, or a recognized user does not haveauthority for the operation of the home appliance 100 corresponding tothe user voice, the processor 140 may not perform the operation of thehome appliance 100 corresponding to the user voice.

The detailed operation in this regard will be described later withreference to FIGS. 9 and 10.

The communication device 170 performs communication with various typesof external devices according to various types of communication methods.Specifically, the communication device 170 may perform communicationwith another home appliance, or perform communication with an externalnetwork (e.g., the Internet, a server, etc.). For such operations, thecommunication device 170 may include a Wi-Fi chip and a Bluetooth chip.

For example, the communication device 170 may perform communication withanother home appliance in an adjacent location by using a Bluetoothchip, or perform communication with an external server through a Wi-Fichip.

In case the home appliance 100 performs a voice recognition function byusing an external server, the communication device 170 may transmitvoice data regarding a voice input through the microphone 120 to anexternal server, and receive the result of voice recognition from theexternal server.

Meanwhile, in case the home appliance 100 performs a speaker recognitionfunction by using an external server, the communication device 170 maytransmit voice data regarding a voice input through the microphone 120to an external server, and receive the result of speaker recognitionfrom the external server.

Then, the processor 140 may identify whether the user has authority byusing the received result of speaker recognition (i.e., user informationcorresponding to the voice data) and pre-stored authority informationregarding a plurality of operations of the home appliance 100 by each ofa plurality of users, and if authority is identified, the processor 140may perform a function corresponding to the result of voice recognition.

The display 180 may display various kinds of information provided at thehome appliance 100. Specifically, the display 180 may display theoperation state of the home appliance 100, or display a user interfacewindow for selection of an operation selected by a user and an option.

For example, the display 180 may display a result of performing a voicerecognition function, or display an interface window so that a user canchange the setting for a voice recognition function.

In the past, a home appliance operated in the voice recognition modeonly when a trigger command was included in a user voice input through amicrophone, and thus there was a problem that a user necessarily had toutter a trigger command before uttering a voice command.

In contrast, as described above, a home appliance according to anembodiment of the disclosure operates in the voice recognition mode if auser action by which it is predicted that a user is going to utter avoice command is detected, even if a user does not utter a triggercommand, and accordingly, the home appliance has an effect of resolvinginconvenience that a user necessarily has to utter a starting command.

Also, a home appliance according to an embodiment of the disclosureoutputs a voice message informing that the voice recognition mode isinitiated when a user action is detected, and can thereby specificallyinform that the home appliance operates in the voice recognition modeeven when it is uncertain whether a user intended the voice recognitionmode of the home appliance, and induce a user to utter a control commandor an inquiry.

In addition, a home appliance according to an embodiment of thedisclosure turns off the microphone in a standby state if an eventwherein a trigger command is detected is inactivated, and turns off thesensor in a standby state if an event wherein a user action is detectedis inactivated, and can thereby reduce power consumed for unnecessarycomponents in a standby state.

FIG. 4 is a block diagram illustrating a voice recognition moduleaccording to an embodiment of the disclosure.

Operations in the voice recognition mode according to an embodiment ofthe disclosure may be performed by using the voice recognition module300 as illustrated in FIG. 4. According to an embodiment of thedisclosure, the voice recognition module 300 may be stored inside thememory 160 of the home appliance 100. However, this is merely anexample, and at least one of the modules included in the voicerecognition module 300 may be included in at least one server outside.

The voice recognition module 300 may include an automatic speechrecognition (ASR) module 310, a natural language understanding (NLU)module 320, a dialogue manger (DM) module 330, a natural languagegenerator (NLG) module 340, and a text to speech (TTS) module 350, asillustrated in FIG. 4.

The automatic speech recognition module (ASR) module 310 may convert auser voice received from the home appliance 100 to text data. Also, theautomatic speech recognition module 310 may include an utterancerecognition module, and the utterance recognition module may include anacoustic model and a language model. In particular, the acoustic modelmay acquire information regarding the acoustic characteristics of a uservoice.

The natural language understanding module 320 may identify a user intentby performing syntactic analysis or semantic analysis.

Also, the natural language understanding module 320 may performsyntactic analysis by dividing a user input into syntactic units (e.g.:words, phrases, morphemes, etc.), and identifying which syntacticelements the divided units have.

In addition, the natural language understanding module 320 may performsemantic analysis by using semantic matching, rule matching, formulamatching, etc. Accordingly, the natural language understanding module320 may acquire a domain which a user input belongs to, an intent, or aparameter (or, a slot) necessary for expressing an intent.

Further, the natural language understanding module 320 may determine auser intent and a parameter by using matching rules divided into adomain, an intent, and a parameter (or, a slot) necessary foridentifying an intent.

For example, a domain (e.g.: an alarm) may include a plurality ofintents (e.g.: setting of an alarm, release of an alarm, etc.), and oneintent may include a plurality of parameters (e.g.: the time, the numberof times of repetition, an alarming sound, etc.). Also, a plurality ofrules may include one or more essential element parameters. Meanwhile,matching rules may be stored in a natural language understandingdatabase (NLU DB) 323.

The natural language understanding module 320 may identify the meaningof words extracted from a user input by using linguistic characteristics(e.g.: syntactic elements) such as morphemes, phrases, etc., and matchthe identified meaning of the words with a domain and an intent anddetermine a user intent.

For example, the natural language understanding module 320 may determinea user intent by calculating how many of the words extracted from a userinput are included in each domain and intent. According to an embodimentof the disclosure, the natural language understanding module 320 maydetermine the parameter of a user input by using words which became abasis for identifying an intent.

Also, the natural language understanding module 320 may determine a userintent by using the natural language understanding database 323 storinglinguistic characteristics for identifying the intent of a user input.

In addition, the natural language understanding module 320 mayunderstand a user inquiry by using a private knowledge DB 325. Here, theprivate knowledge DB 325 may learn the relation among knowledgeinformation based on at least one of a user interaction input into thehome appliance 100, sensing information sensed by the home appliance100, or user information received from an external device. Here, theprivate knowledge DB 325 may store the relation among knowledgeinformation in the form of ontology.

The conversation manager module 330 may determine whether a user intentidentified by the natural language understanding module 320 is clear.For example, the conversation manager module 330 may determine whether auser intent is clear based on whether there is sufficient information ona parameter. Also, the conversation manager module 330 may determinewhether a parameter identified at the natural language understandingmodule 320 is sufficient for performing a task.

According to an embodiment of the disclosure, the conversation managermodule 330 may perform a feedback requesting necessary information to auser in case a user intent is not clear. For example, the conversationmanager module 330 may perform a feedback requesting information for aparameter for identifying a user intent. Also, the conversation managermodule 330 may generate a message for identifying a user inquiryincluding a text changed by the natural language understanding module320 and output the message.

The natural language generator (NLG) module 340 may change designatedinformation into the form of a text. The information changed into theform of a text may be in the form of utterance of a natural language.Here, designated information may be information regarding an additionalinput, information guiding completion of an operation corresponding to auser input, or information guiding an additional input of a user (e.g.:feedback information for a user input).

The text to speech (TTS) module 350 may change information in the formof a text into information in the form of a voice. The text to speechmodule 350 may receive information in the form of a text from thenatural language generator module 340, and change the information in theform of a text into information in the form of a voice and output theinformation through the speaker. Also, the information in the form of atext may be displayed on the display 180 of the home appliance 100.

FIGS. 5 and 6 are diagrams for illustrating various methods forinitiating a voice recognition mode.

If a user action is detected through the sensor 110, the home appliance100 may operate in the voice recognition mode. For example, if openingor closing of the door is detected through the sensor 110, the homeappliance 100 may operate in the voice recognition mode.

Hereinafter, it will be assumed that the home appliance 100 is a washingmachine for the convenience of explanation, and the operations of thehome appliance 100 will be explained with respect to each of the caseswherein a user action is opening or closing of the door.

FIG. 5 is a diagram for illustrating a method for initiating a voicerecognition mode according to the first embodiment of the disclosure. Inthis embodiment, a case wherein a user action is set as opening of thedoor is assumed.

If a user accesses a washing machine and utters a trigger command (e.g.,“Hi, Bixby”), the home appliance 100 may operate in the voicerecognition mode. Then, the home appliance 100 may output an audiosignal corresponding to an event of detecting a trigger command. Here,the output audio signal is a simple alarming sound informing that thevoice recognition mode was initiated. Referring to FIG. 5, an audiosignal corresponding to an event of detecting a trigger command may be asimple alarming sound {circle around (1)} like “ding-dong.”

Meanwhile, the user may open the door for putting laundry into the drumof the washing machine. In this case, the home appliance 100 may detectopening of the door and operate in the voice recognition mode. Then, thehome appliance 100 may output an audio signal corresponding to an eventof opening of the door. Here, the output audio signal is a voice messagespecifically informing that the voice recognition mode was initiated.Referring to FIG. 5, an audio signal corresponding to an event ofopening of the door may be a voice message {circle around (2)} like“What can I help you with?”. Meanwhile, a voice message is not limitedto the aforementioned example, and it may be a voice message like “Thevoice recognition mode is initiated.”

Then, the home appliance 100 may perform voice recognition for the uservoice input while operating in the voice recognition mode, and performan operation corresponding to the result of voice recognition.

Meanwhile, other than a manner of operating in the voice recognitionmode if an event wherein opening of the door is detected occurs, thehome appliance 100 may also be implemented in a manner of operating inthe voice recognition mode if an event wherein closing of the door isdetected occurs. Explanation in this regard will be made below.

FIG. 6 is a diagram for illustrating a method for initiating a voicerecognition mode according to the second embodiment of the disclosure.In this embodiment, a case wherein a user action is set as closing ofthe door is assumed.

Referring to FIG. 6, a user may open the door to put laundry into thedrum of the washing machine. However, as opening of the door was not setas a user action, the home appliance 100 may maintain the standby statewithout operating in the voice recognition mode.

Then, when the user closes the door of the washing machine again, thehome appliance 100 may detect closing of the door and operate in thevoice recognition mode. Also, the home appliance 100 may output an audiosignal corresponding to an event of closing of the door. Here, theoutput audio signal is a voice message specifically informing that thevoice recognition mode is initiated.

As described above, the home appliance 100 may vary the method forinitiating the voice recognition mode according to whether an action setas a user action is opening or closing of the door.

Meanwhile, a user action may be set at the time of manufacture. Also, auser action may be set as opening or closing of the door according to auser setting. Specifically, the home appliance 100 may receive input ofat least one operation of opening or closing of the door as a useraction from a user through the user interface device 150. In this case,the home appliance 100 may set the input operation as a user action, andif the input action is detected, the home appliance 100 may operate inthe voice recognition mode.

Meanwhile, if a specific condition is satisfied after the home appliance100 operates in the voice recognition mode, the home appliance 100 maybe converted to the standby mode again. Detailed explanation in thisregard will be made below.

FIG. 7 is a diagram for illustrating a method for converting to astandby mode according to the first embodiment of the disclosure. Inthis embodiment, a case wherein a user action is set as opening of thedoor is assumed.

Referring to FIG. 7, it can be seen that the user closed the door of thewashing machine. When opening of the door is detected, the homeappliance 100 may operate in the voice recognition mode, and whenclosing of the door is detected afterwards, the home appliance 100 maybe converted to the standby mode again.

Specifically, if closing of the door is detected while the homeappliance 100 performs a voice recognition function based on opening ofthe door, the home appliance 100 may identify that the user will notutter a voice command anymore, and may be converted from the voicerecognition mode to the standby state.

Meanwhile, the home appliance 100 may maintain the voice recognitionmode even if closing of the door is detected. Explanation in this regardwill be made below with reference to FIG. 8.

FIG. 8 is a diagram for illustrating a method for converting to astandby mode according to the second embodiment of the disclosure.

The home appliance 100 may maintain the voice recognition mode accordingto whether a control command or an inquiry of a user is included in avoice input through the microphone 120 while operating in the voicerecognition mode.

In case a control command or an inquiry of a user is included in a voiceinput through the microphone 120 while operating in the voicerecognition mode, the home appliance 100 may maintain the voicerecognition mode and perform an interaction with the user.

For example, referring to FIG. 8, a user may utter a control commandsuch as “Wash the clothes” while the door of the washing machine isopened, and then close the door of the washing machine. In this case, asa control command of a user is included in a voice input through themicrophone 120 while operating in the voice recognition mode, the homeappliance 100 may maintain the voice recognition mode. Also, the homeappliance 100 may perform an interaction with a user by outputting avoice such as “How long will I wash the clothes?”.

In contrast, in case a control command or an inquiry of a user is notincluded in a voice input through the microphone 120 while operating inthe voice recognition mode, the home appliance 100 may identify that theuser does not intend to use a voice recognition function, and may beconverted from the voice recognition mode to the standby state.

Meanwhile, in illustrating and describing FIGS. 7 and 8, it wasillustrated and described that the user action is opening of the door,but the embodiments can also be applied in the same manner to a casewherein a user action is closing of the door.

Specifically, in case a user action is closing of the door, the homeappliance 100 may operate in the voice recognition mode, and may then beconverted from the voice recognition mode to the standby state accordingto whether a predetermined time has been reached.

Meanwhile, the home appliance 100 may maintain the voice recognitionmode according to whether a control command or an inquiry of a user isincluded in a voice input through the microphone 120 for a predeterminedtime period.

In case a control command or an inquiry of a user is included in a voiceinput through the microphone 120 for a predetermined time period, thehome appliance 100 may maintain the voice recognition mode and performan interaction with a user.

In contrast, in case a control command or an inquiry of a user is notincluded in a voice input through the microphone 120 for a predeterminedtime period, the home appliance 100 may identify that the user does notintend to use a voice recognition function and may be converted from thevoice recognition mode to the standby state.

As described above, if a predetermined event is detected, the homeappliance 100 may operate in the voice recognition mode, and if apredetermined condition is satisfied, the home appliance 100 may beconverted from the voice recognition mode to the standby state again,and accordingly, a circumstance of misrecognition and power consumptionaccording to maintaining the voice recognition mode consistently can beprevented.

Meanwhile, in illustrating and describing FIGS. 5 to 8, it was assumedthat the home appliance is a washing machine, but the type of the homeappliance is not limited to a washing machine, and it may be any type ifit is a home appliance which includes a door and a sensor detectingopening and closing of the door.

Meanwhile, even if the home appliance 100 operates in the voicerecognition mode and receives input of a control command from a user asin the aforementioned method, the home appliance 100 may perform acorresponding operation according to whether the user has authority foran operation corresponding to the control command. A detailed operationin this regard will be described below.

FIG. 9 is a diagram illustrating examples of authority information ofeach of a plurality of users. If a user voice is input through themicrophone 120, the processor 140 may identify the user voice andrecognize the user, and if an operation of the home appliance 100corresponding to the user voice matches the authority information of therecognized user, the processor 140 may perform the operation of the homeappliance 100 corresponding to the user voice.

To be more specific, the processor 140 may recognize a user by using auser voice input through the microphone 120 in the voice recognitionmode and the speaker recognition model stored in the memory 160.

Then, the processor 140 may identify whether the operation of the homeappliance 100 corresponding to the user voice (e.g., a washingoperation) matches the authority information of the recognized user byusing the authority information of the user stored in the memory 160,and if the authority matches, the processor 140 may perform theoperation of the home appliance 100 corresponding to the user voice.

For example, referring to FIG. 9, in case the home appliance 100 is anoven, the home appliance 100 may store authority information for thefather, the mother, and the baby who are a plurality of users. Accordingto the illustrated authority information, it can be figured out that thefather and the mother have authority for a preheating operation and abaking operation, but the baby does not have authority for a preheatingoperation and a baking operation.

Accordingly, the processor 140 may identify who the current speaker isamong the father, the mother, and the baby in the voice recognitionmode, and identify whether the identified user has the authority for theoperation corresponding to the control command, and if the user has theauthority, the processor 140 may perform the operation, and if the userdoes not have the authority, the processor 140 may not perform theoperation.

FIG. 10 is a diagram for illustrating execution of operations accordingto authorization information of each of a plurality of users.

According to an embodiment of the disclosure, execution of an operationaccording to authority information of a user as described above may beperformed while being limited to a case wherein a control commandidentified as a result of voice recognition is a predeterminedoperation. Here, a predetermined operation may be an operation requiringsafety in performing the operation. For example, in case the homeappliance 100 is an oven, the predetermined operation may be a bakingoperation or a preheating operation.

Specifically, in case an operation corresponding to a user voice inputthrough the microphone 120 is a predetermined operation, the processor140 may recognize which user the current speaker is among pre-registeredusers, and identify whether the recognized user has authority for thepredetermined operation.

For example, referring to FIG. 10, it can be seen that the baby uttered“Bake it for one hour” {circle around (1)}. In this case, the processor140 may identify that the control command included in the uttered voiceis a baking operation, and is an operation requiring safety. Then, theprocessor 140 may identify the input voice and recognize that thecurrent speaker is the baby, and identify whether the baby has authorityfor the baking operation.

However, as the baby does not have authority for the baking operation,the processor 140 may not execute the baking operation. Then, forguiding that the baking operation cannot be performed, the processor 140may control the speaker 130 to output a voice like “It is dangerous” E.Alternatively, the processor 140 may display a message guiding that thebaking operation cannot be executed through the display 180.

Meanwhile, the processor 140 may receive input of authority for aplurality of operations of the home appliance 100 for each ofpre-registered users from a user. For example, with respect to “thebaby” who is a pre-registered user, the processor 140 may receive inputof a command of granting authority or eliminating authority for thebaking operation among the operations of the home appliance 100 throughthe user interface device 150.

Meanwhile, the operations of the home appliance 100 are not limited tothe aforementioned baking operation and preheating operation, and mayinclude various operations according to the characteristics of the homeappliance 100.

As described above, the processor performs an operation corresponding toa control command by using authority information of each of a pluralityof users, and thus there can be effects of safe use of the homeappliance and maintenance of security.

FIG. 11 is a diagram for illustrating an algorithm of a voicerecognition mode according to an embodiment of the disclosure.

Referring to FIG. 11, the processor 140 of the home appliance 100 maydetect whether a voice including a trigger command is input through themicrophone 120 in a standby state at operation S1110.

If a trigger command is detected at operation S1110-Y, the processor 140may output an audio signal corresponding to the event wherein a triggercommand was detected at operation S1130. Here, an audio signalcorresponding to the event wherein a trigger command was detected may bea simple alarming sound.

In contrast, if a trigger command is not detected at operation S1110-N,the processor 140 may identify whether a user action is detected throughthe sensor 110 at operation S1120.

If a user action is detected at operation S1120-Y, the processor 140 mayoutput an audio signal corresponding to the event wherein a user actionwas detected at operation S1135. Here, an audio signal corresponding tothe event wherein a user action was detected may be a voice messageinforming that the voice recognition mode is initiated.

In contrast, if a user action is not detected at operation S1120-N, theprocessor 140 may keep maintaining the standby state, and detect whethera voice including a trigger command is input through the microphone 120at operation S1110.

Meanwhile, after outputting an alarming sound or a voice message, theprocessor 140 may operate in the voice recognition mode at operationS1140. Specifically, the processor 140 may perform voice recognition fora voice input through the microphone 120 by using the voice recognitionmodule.

Then, the processor 140 may identify whether the operation correspondingto the result of voice recognition is a predetermined operation atoperation S1150.

In case the operation corresponding to the result of voice recognitionis not a predetermined operation at operation S1150-N, the correspondingfunction is an operation for which determination of authority of theuser is unnecessary. Thus, the processor 140 may execute the operationcorresponding to the result of voice recognition at operation S1180.

In contrast, in case the operation corresponding to the result of voicerecognition is a predetermined operation at operation S1150-Y, theprocessor 140 may recognize the user of the voice input through themicrophone 120. Specifically, the processor 140 may recognize the userby identifying the input voice by using the speaker recognition model.

Then, the processor 140 may identify whether the recognized user is apre-registered user at operation S1160.

In case the recognized user is a pre-registered user at operationS1160-Y, the processor 140 may identify whether the recognized user hasauthority for the operation corresponding to the result of voicerecognition at operation S1170.

In case the recognized user has authority for the operationcorresponding to the result of voice recognition, the processor 140 mayexecute the operation corresponding to the result of voice recognitionat operation S1180.

In case the recognized user is not a pre-registered user at operationS1160-N, or the recognized user does not have authority for theoperation corresponding to the result of voice recognition at operationS1170-N, the processor 140 may not perform the operation correspondingto the result of voice recognition, and may be converted to the standbystate.

Meanwhile, in illustrating and describing FIG. 11, it was illustratedand described that it is identified whether a trigger command isdetected, and then it is identified whether a user action is detected.However, in actual implementation, implementation may be made in a waythat it is identified whether a user action is detected, and then it isidentified whether a trigger command is detected.

Also, in illustrating and describing FIG. 11, it was illustrated anddescribed that an alarming sound or a voice message is output, and thenthe processor operates in the voice recognition mode. However, in actualimplementation, implementation may be made in a way that the processoroperates in the voice recognition mode, and then an alarming sound or avoice message is output.

In addition, in illustrating and describing FIG. 11, it was illustratedand described that the processor of the home appliance performs all ofthe aforementioned operations. However, in actual implementation,implementation may be made in a way that a server performs some of theaforementioned operations of the processor. Detailed explanation in thisregard will be made below with reference to FIGS. 12 and 13.

FIG. 12 is a sequence diagram for illustrating execution of operationsaccording to authorization information of each of a plurality of usersaccording to the first embodiment of the disclosure.

First, the home appliance 100 may detect a predetermined event atoperation S1210. Specifically, the home appliance 100 may detect anevent wherein a trigger command is input through the microphone 120 oran event wherein a user action is detected through the sensor 110.

Then, the home appliance 100 may operate in the voice recognition modewhen a predetermined event is detected at operation S1220. Specifically,the home appliance 100 may output an audio signal corresponding to eachpredetermined event, and receive input of a user voice through themicrophone 120.

Then, the home appliance 100 may transmit voice data corresponding tothe input voice to the server 200 at operation S1230.

Then, the server 200 may recognize a user corresponding to the speakerby using the received voice data at operation S1240. Specifically, theserver 200 may recognize a user corresponding to the speaker by usingthe speaker recognition model.

To be more specific, the server 200 may extract a vocal print from thereceived voice data. Then, the server 200 may compare the plurality ofextracted vocal print information with the vocal print information ofpre-registered users and identify whether the speaker corresponds to apre-registered user.

Then, the server 200 may transmit information for the recognized user tothe home appliance 100 at operation S1250.

Next, the home appliance 100 may identify an operation corresponding tothe voice input through the microphone 120 at operation S1260.Specifically, the home appliance 100 may perform voice recognition forthe voice input through the microphone 120 by using the voicerecognition module, and identify the control command, and identify theoperation of the home appliance 100 corresponding to the controlcommand.

Then, with respect to the operation of the home appliance 100corresponding to the identified control command, the home appliance 100may identify whether the recognized user has authority at operationS1270. Specifically, the home appliance 100 may identify whether therecognized user has authority for the operation corresponding to theidentified control command by using authority information of each of aplurality of users.

Then, the home appliance 100 may provide a feedback according to whetherthe recognized user has authority at operation S1280. Specifically, incase the recognized user has authority, the home appliance 100 mayexecute the operation corresponding to the control command.

In contrast, in case the recognized user does not have authority or therecognized user is not a pre-registered user, the home appliance 100 maynot execute the operation corresponding to the control command. In thiscase, the home appliance 100 may provide guidance regardingimpossibility of execution of the operation to the user. For example,the home appliance 100 may output a predetermined guidance message soundor display a predetermined guidance message through the display 180.

In case a home appliance does not include a speaker recognition model asabove, the home appliance may recognize a user corresponding to an inputvoice by using a server that can perform a speaker recognition function.

Meanwhile, a home appliance may not include a voice recognition moduleas well as a speaker recognition model depending on its kind or usage.Even in this case, speaker recognition and voice recognition may beperformed by using a server. Explanation in this regard will be madebelow.

FIG. 13 is a sequence diagram for illustrating execution of operationsaccording to authorization information of each of a plurality of usersaccording to the second embodiment of the disclosure.

First, the home appliance 100 may detect a predetermined event atoperation S1310. Then, when a predetermined event is detected, the homeappliance 100 may operate in the voice recognition mode at operationS1320. Specifically, the home appliance 100 may output an audio signalcorresponding to each predetermined event, and receive input of a uservoice through the microphone 120.

Then, the home appliance 100 may transmit voice data corresponding tothe input voice to the server 200 at operation S1330.

Then, the server 200 may recognize a user corresponding to the speakerby using the received voice data at operation S1340. Specifically, theserver 200 may recognize a user corresponding to the speaker by usingthe speaker recognition model.

Next, the server 200 may identify an operation of the home appliance 100corresponding to the received voice data at operation S1350.Specifically, the server 200 may perform voice recognition for thereceived voice data by using the voice recognition module, and identifythe control command, and identify the operation of the home appliance100 corresponding to the control command.

Then, the server 200 may transmit information for the recognized userand information on the identified operation to the home appliance 100 atoperation S1360.

Also, with respect to the identified operation, the home appliance 100may identify whether the recognized user has authority at operationS1370. Specifically, the home appliance 100 may identify whether therecognized user has authority for the identified operation by usingauthority information of each of a plurality of users.

Then, the home appliance 100 may provide a feedback according to whetherthe recognized user has authority at operation S1380. Specifically, incase the recognized user has authority, the home appliance 100 mayexecute the identified operation.

In contrast, in case the recognized user does not have authority or therecognized user is not a pre-registered user, the home appliance 100 maynot execute the identified operation. In this case, the home appliance100 may provide guidance regarding impossibility of execution of theoperation to the user. For example, the home appliance 100 may output apredetermined guidance message sound or display a predetermined guidancemessage through the display 180.

Even in a case wherein a home appliance does not include a speakerrecognition model and a voice recognition module as above, the homeappliance may recognize a user corresponding to an input voice by usinga server that can perform a speaker recognition function and a voicerecognition function, and identify an operation corresponding to theinput voice.

Meanwhile, in illustrating and describing FIG. 13, it was illustratedand described that the home appliance identifies authority informationfor an operation of the recognized user. However, in actualimplementation, implementation may be made in a way that a serverperforms the aforementioned operation, and transmits the result ofdetermination on whether the recognized user has authority to the homeappliance.

FIG. 14 is a flow chart for illustrating a method for a home applianceto recognize a voice according to an embodiment of the disclosure.

Referring to FIG. 14, if one of a first event wherein a user action isdetected or a second event wherein a trigger command for initiating thevoice recognition mode occurs, an audio signal corresponding to theevent occurred is output at operation S1410.

Specifically, it may be identified whether the first event occurredthrough a sensor detecting a user action. Here, a user action may be setdifferently according to the type of the sensor detecting the useraction.

For example, in case the sensor is a sensor for detecting opening andclosing of the door provided on the main body of the home appliance, theuser action may be set as opening or closing of the door. Accordingly,if the door is opened or the door is closed, it may be determined thatthe first event occurred.

As another example, in case the sensor is a motion detection sensor, theuser action may be a user access to the home appliance. Accordingly, ifa user access to the home appliance is detected, it may be determinedthat the first event occurred.

Meanwhile, it may be identified whether the second event occurredaccording to whether a trigger command is included in a voice inputthrough the microphone of the home appliance. Here, a trigger commandmay be referred to as a wake-up command or a starting command, etc.

Also, an audio signal output may be set differently for each of thefirst event and the second event.

To be more specific, an audio signal corresponding to the first eventmay be a voice message informing that the voice recognition mode isinitiated. For example, an audio signal corresponding to the first eventmay be a voice message specifically informing that the voice recognitionmode is initiated like “What can I help you with?”. Meanwhile, the typeof the voice message is not limited to the aforementioned example.

Also, an audio signal corresponding to the second event may be a simplealarming sound. For example, an audio signal corresponding to the secondevent may be a simple alarming sound like “ding-dong.” Meanwhile, thetype of the alarming sound is not limited to the aforementioned example.

Then, the home appliance operates in the voice recognition mode atoperation S1420. Here, the voice recognition mode means an operationmode wherein a user's inquiry and response input through the microphoneare processed as a natural language. Here, the voice recognition modemay be implemented in a manner of directly processing a user's inquiryand response as a natural language by using a voice recognition module.Also, the voice recognition mode may be implemented in a manner oftransmitting a voice input through the microphone to a server andreceiving the result of voice recognition from the server.

Specifically, voice recognition may be performed for a voice inputthrough the microphone after the home appliance starts to operate in thevoice recognition mode.

Then, an operation corresponding to the result of voice recognition maybe performed. Meanwhile, before performing the operation correspondingto the result of voice recognition, the authority of the user may beidentified, and the operation may be performed according to whether theuser has authority.

Specifically, first, when a user voice is input in the voice recognitionmode, the user voice may be identified and the user may be recognized.To be more specific, the user may be recognized by using the user voiceinput in the voice recognition mode and the speaker recognition modelstored in the memory.

Then, the authority information of the recognized user may be read fromthe memory. If the operation corresponding to the user voice matches theread authority information, the operation corresponding to the uservoice (i.e., the operation corresponding to the result of voicerecognition) may be executed.

In contrast, if the operation corresponding to the user voice does notmatch the read authority information, or a user corresponding to theinput user voice is not identified, the operation corresponding to theuser voice may not be executed.

Meanwhile, a decision regarding activation may be input for each of thefirst event and the second event. Specifically, a decision regardingactivation may be input for each of the first event and the second eventthrough the user interface device provided on the home appliance.

If inactivation of the second event is input through the user interfacedevice, the microphone of the home appliance may be turned off in astandby state. Here, a standby state means a state wherein it isdetected whether a predetermined event occurred to initiate an operationin the voice recognition mode.

Then, when the first event occurs afterwards, the microphone may beturned on again. As described above, if inactivation of the second eventis input, there is no need to detect a trigger command. Accordingly, themicrophone is turned off, and power consumption can thereby beprevented.

In contrast, if inactivation of the first event is input through theuser interface device, the microphone may be maintained in the turned-onstate in the standby state and the sensor detecting a user action may beturned off. As described above, if inactivation of the first event isinput, there is no need to detect a user action. Accordingly, the sensoris turned off, and power consumption can thereby be prevented.

Meanwhile, implementation may be made such that the subject detecting atrigger command and the subject operating in the voice recognition modeare different from each other. Specifically, the home appliance mayinclude a first processor detecting a trigger command and a secondprocessor for operating in the voice recognition mode.

Also, the first processor may maintain the turned-on state in thestandby state. In addition, the first processor may recognize a triggercommand included in a user voice.

Further, when the first processor recognizes a trigger command, thefirst processor may turn on the second processor. Specifically, if thestandby state of the second processor is maintained for over apredetermined time period, the state of the second processor may bechanged from a turned-on state to a turned-off state and the state maybe maintained. Also, when the first processor recognizes a triggercommand, the first processor may turn on the second processor.

Accordingly, in the method for controlling a home appliance according tothe disclosure, even if a user does not utter a trigger command, if auser action by which it is predicted that a user is going to utter avoice command is detected, a home appliance operates in the voicerecognition mode. Accordingly, there is an effect that the inconveniencethat a user necessarily has to utter a starting command is resolved.Also, when a user action is detected, a voice message informing that thevoice recognition mode is initiated is output, and accordingly, it canbe specifically informed that a home appliance will operate in the voicerecognition mode even when it is uncertain whether the user intended thevoice recognition mode of the home appliance, and the user may beinduced to utter a control command or an inquiry. In addition, if anevent wherein a trigger command is detected is inactivated, themicrophone is turned off in the standby state, and if an event wherein auser action is detected is inactivated, the sensor is turned off in thestandby state, and accordingly, power consumed for unnecessarycomponents in the standby state can be reduced.

The method for recognizing a voice as in FIG. 14 may be executed in ahome appliance having the configurations as in FIG. 2 or FIG. 3, and mayalso be executed in a home appliance having a different configuration.

Also, the aforementioned method for controlling a home appliance may beimplemented as at least one execution program for executing theaforementioned method for controlling a home appliance, and such anexecution program may be stored in a non-transitory computer readablemedium.

A non-transitory computer readable medium refers to a medium that storesdata semi-permanently, and is readable by machines, but not a mediumthat stores data for a short moment such as a register, a cache, and amemory. Specifically, the aforementioned various applications orprograms may be provided while being stored in a non-transitory computerreadable medium such as a CD, a DVD, a hard disk, a blue-ray disk, aUSB, a memory card, a ROM and the like.

Also, while preferred embodiments of the disclosure have been shown anddescribed, the disclosure is not limited to the aforementioned specificembodiments, and it is apparent that various modifications can be madeby those having ordinary skill in the art to which the disclosurebelongs, without departing from the gist of the disclosure as claimed bythe appended claims, and such modifications are within the scope of thedescriptions of the claims.

What is claimed is:
 1. A home appliance comprising: a proximity sensorto detect a user of the home appliance in proximity of the homeappliance; a microphone to receive an utterance of a trigger command ofthe user; a user interface through which it is settable whether a voicerecognition mode is to be activated in response to the proximity sensordetecting the user in proximity of the home appliance, and whether thevoice recognition mode is to be activated in response to the microphonereceiving the utterance of the trigger command of the user; and at leastone processor configured to: perform a first operation of activating thevoice recognition mode in response to the proximity sensor detecting theuser in proximity of the home appliance, when the voice recognition modeis set through the user interface to be activated in response to theproximity sensor detecting the user in proximity of the home appliance,and, with the voice recognition mode activated by the first operation,inactivate the voice recognition mode in response to a predeterminedevent occurring while the proximity sensor continues to detect the userin proximity of the home appliance.
 2. The home appliance as in claim 1,wherein the at least one processor is configured not to perform thefirst operation of activating the voice recognition mode when the voicerecognition mode is set through the user interface to not be activatedin response to the proximity sensor detecting the user in proximity ofthe home appliance.
 3. The home appliance as in claim 2, wherein the atleast one processor is configured to turn off the microphone when thevoice recognition mode is set through the user interface to not to beactivated in response to the microphone receiving the utterance of thetrigger command of the user.
 4. The home appliance as in claim 1,wherein the predetermined event is a lapse of a predetermined amount oftime from the activation of the voice recognition mode in the firstoperation.
 5. The home appliance as claimed in claim 1, wherein the atleast one processor is configured to: perform a second operation ofactivating the voice recognition mode in response to the microphonereceiving the utterance of the trigger command of the user, when thevoice recognition mode is set through the user interface to be activatedin response to the microphone receiving the utterance of the triggercommand of the user.
 6. The home appliance as claim in claim 5, whereinthe at least one processor is configured to: output a first notificationto the user when the first operation of activating the voice recognitionmode is performed, and output a second notification, different from thefirst notification, to the user when the second operation of activatingthe voice recognition mode is performed.
 7. A home appliance comprising:a main body including a door; a proximity sensor to detect a user inproximity of the home appliance; a microphone to receive an utterance ofthe user; a user interface through which it is settable whether a voicerecognition mode is to be activated in response to detecting the user inproximity of the home appliance, and whether the voice recognition modeis to be activated in response to a trigger command included in theutterance of the user; and at least one processor configured to: performa first operation of activating the voice recognition mode in responseto detecting the user in proximity of the home appliance via theproximity sensor, while the voice recognition mode is set through theuser interface to be activated in response to detecting the user inproximity of the home appliance, perform a second operation ofactivating the voice recognition mode in response to the trigger commandincluded in the utterance of the user received via the microphone, whilethe voice recognition mode is set through the user interface to beactivated in response to the trigger command included in the utteranceof the user, and not perform the first operation of activating the voicerecognition mode regardless of detecting the user in proximity of thehome appliance via the proximity sensor, while the voice recognitionmode is set through the user interface not to be activated in responseto detecting the user in proximity of the home appliance.
 8. The homeappliance as in claim 7, wherein the at least one processor isconfigured to turn off the microphone when the voice recognition mode isset through the user interface not to be activated in response inresponse to the trigger command included in the utterance of the user.9. The home appliance as in claim 7, wherein the at least one processoris configured to: output a first notification to the user when the firstoperation of activating the voice recognition mode is performed, andoutput a second notification, different from the first notification, tothe user when the second operation of activating the voice recognitionmode is performed.
 10. The home appliance as in claim 9, the secondnotification comprises an alarm sound.
 11. The home appliance as inclaim 7, wherein the at least one processor is configured to, with thevoice recognition mode activated by the first operation, inactivate thevoice recognition mode in response to a predetermined event occurring.12. The home appliance as in claim 11, wherein the predetermined eventis a lapse of a predetermined amount of time from the activation of thevoice recognition mode in the first operation.
 13. The home appliance asin claim 11, wherein the predetermined event is the door being opened.14. A method for controlling a home appliance, the method comprising:receiving a user input, through a user interface of the home appliance,for setting whether a voice recognition mode is to be activated inresponse to detecting a user in proximity of the home appliance;detecting a user in proximity of the home appliance via a proximitysensor of the home appliance; detecting a trigger command included in anutterance of the user received via a microphone; performing a firstoperation of activating the voice recognition mode in response todetecting the user in proximity of the home appliance via the proximitysensor when the voice recognition mode is set through the user interfaceto be activated in response to detecting the user in proximity of thehome appliance, performing a second operation of activating the voicerecognition mode in response to detecting the trigger command include inthe utterance of the user received via the microphone, not performingthe first operation regardless of detecting the user in proximity of thehome appliance via the proximity sensor when the voice recognition modeis set through the user interface not to be activated in response todetecting the user in proximity of the home appliance, outputting afirst notification to the user when the first operation is performed,and outputting a second notification, different from the firstnotification, to the user when the second operation is performed. 15.The method as in claim 14, further comprising: inactivating the voicerecognition mode in response to a predetermined event occurring afterthe activation of the voice recognition mode in the first operation. 16.The method as in claim 15, wherein the predetermined event is a lapse ofa predetermined amount of time from the activation of the voicerecognition mode in the first operation.
 17. The method as in claim 15,the predetermined event is a door of the home appliance being openedafter the activation of the voice recognition mode in the firstoperation.
 18. The method as in claim 17, the second notificationcomprises an alarm sound.